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Abstract 

We address the problem of learning a joint model of 
actors and actions in movies using weak supervision pro¬ 
vided by scripts. Specifically, we extract actor/action pairs 
from the script and use them as constraints in a discrimi¬ 
native clustering framework. The corresponding optimiza¬ 
tion problem is formulated as a quadratic program under 
linear constraints. People in video are represented by au¬ 
tomatically extracted and tracked faces together with cor¬ 
responding motion features. First, we apply the proposed 
framework to the task of learning names of characters in 
the movie and demonstrate significant improvements over 
previous methods used for this task. Second, we explore 
the joint actor/action constraint and show its advantage 
for weakly supervised action learning. We validate our 
method in the challenging setting of localizing and recog¬ 
nizing characters and their actions in feature length movies 
Casablanca and American Beauty. 

1. Introduction 

The recognition of actions, scenes and objects in videos 
is a difficult task due to the large variability of their visual 
appearance. Modeling such a variability typically requires 
manually annotating large numbers of training samples for 
learning model parameters. Video annotation, however, is 
a very tedious process that does not scale well to the huge 
number of existing events. 

Video scripts exist for thousands of movies and TV- 
series and contain rich descriptions in terms of people, their 
actions, interactions and emotions, object properties, scene 
layouts and more. Previous work has explored video scripts 
to learn and automatically annotate characters in TV se¬ 
ries [6, 22, 23]. Automatic learning of human actions from 
scripts has also been attempted [8, 18, 20]. The problem, 
however, remains difficult due to the lack of explicit corre¬ 
spondence between scene elements in video and their tex¬ 
tual descriptions in scripts. In particular, video scripts pro- 
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Rick sits down again and stares off in their 
direction, lisa and Laszlo leave the cafe. 


Figure 1: Result of our automatic detection and annotation 
of characters and their actions in the movie Casablanca. The 
automatically resolved correspondence between video and 
script is color-coded. 

vide no spatial localization of people and objects, and the 
temporal localization of events inferred from the subtitles is 
often imprecise. 

Previous work on weakly supervised learning in im¬ 
ages [5, 13, 19] and video [6, 8, 18, 20, 22, 23] has ex¬ 
plored redundancy to resolve ambiguity of textual annota¬ 
tion. For example, multiple images known to contain a per¬ 
son X could help identifying X by intersecting sets of peo¬ 
ple from each image. Given the difficulty of identifying the 
same person, action or object class in different images or 
videos, the realization of the “intersection” idea, however, 
is often non-trivial in practice. 

Objects, people and actions often co-occur. Knowing 
that “Rick sits down” in a video can help annotating a sit¬ 
ting down action if we can localize Rick and vice versa, 
see Figure 1 . Action recognition can particularly help per¬ 
son identification for rare subjects and subjects facing away 
from the camera (e.g., Ilsa walks away to the door). Rec¬ 
ognizing actors, on the other hand, can be most useful for 
learning rare events (e.g. hand shaking). 

We follow this intuition and address joint weakly super¬ 
vised learning of actors and actions by exploiting their co¬ 
occurrence in movies. We follow previous work [6, 8, 18, 
20, 22, 23] and use movie scripts as a source of weak super¬ 
vision. Differently from this prior work, we use actor-action 
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co-occurrences derived from scripts to constrain the weakly 
supervised learning problem. 

As one of our main contributions, we formulate weakly 
supervised joint learning of actors and actions as an opti¬ 
mization of a new discriminative cost function. We first 
investigate weakly supervised learning of actors only and 
demonstrate the benefit of our learning method in compari¬ 
son with other weakly supervised techniques designed for 
this task [6, 22]. We then demonstrate the advantage of 
the joint constraints for action recognition. We validate our 
method in the challenging setting of localizing and recog¬ 
nizing actors and their actions in movies Casablanca and 
American Beauty. An example output of our algorithm for 
a short movie clip and the associated script section is illus¬ 
trated in Figure 1 . 

Related Work. Learning from images and text has been 
addressed in the context of automatic annotation of images 
with keywords [4, 11, 25] or labeling faces with names in 
news collections [5]. Berg et al. [: ] label detected faces in 
news photographs with names of people obtained from text 
captions. Recent work has looked at learning spatial rela¬ 
tions (such as “on top of’) from prepositions [13] or gen¬ 
erating entire sentence-level captions for images [10, 21]. 
A generative model of faces and poses (such as “Hit Back¬ 
hand”) was learnt from names and verbs in manually pro¬ 
vided captions for news photographs [19]. While the goal of 
this work is related to ours, we focus on learning from video 
with sparse, noisy and imprecise annotations extracted from 
scripts. To deal with the ambiguity of annotations, we de¬ 
velop a new discriminative weakly supervised clustering 
model of video and text. 

In video, manually provided text descriptions have been 
used to learn a causal model of human actions in the con¬ 
strained domain of sports events [14]. Others have looked 
at learning from videos with readily-available text, but 
names [6, 9, 22] and actions [8, 18] have been so far consid¬ 
ered separately. The ambiguity and errors of readily avail¬ 
able annotations present a major challenge for any learning 
algorithm. These problems have been addressed by design¬ 
ing appropriate loss functions [( ] or explicitly finding the 
corresponding instances in video using multiple instance 
learning [ 8 ]. Others have looked at convex relaxations of 
discriminative clustering with hidden variables for image 
co-segmentation [16, 17]. 

Contributions. First, we consider a richer use of textual 
information for video and learn from pairs of names and 
actions co-occurring in the text. Second, we formulate the 
problem of finding characters and their actions as weakly 
supervised structured classification of pairs of action and 
name labels. Third, we develop a new discriminative clus¬ 
tering model jointly learning both actions and names and in¬ 


corporating text annotations as constraints. The correspond¬ 
ing optimization is formulated as a quadratic program under 
linear constraints. Finally, we demonstrate the validity of 
the model on two feature-length movies and corresponding 
movie scripts, and demonstrate improvements over earlier 
weakly supervised methods. 

2. Joint Model of Actors and Actions 

We formulate the problem of jointly detecting actors and 
actions as discriminative clustering [2, 17]: grouping sam¬ 
ples into classes so that an appropriate loss is minimized. 
We incorporate text-based knowledge as a suitable set of 
constraints on the cluster membership. 

2.1. Notation 

Let us suppose that we have two label sets V and A and 
that \P\ = P and \A\ = A. In practice, one can think of 
these as person and action classes. 

Our data is organized into sets, that we refer to as bags, 
and which are indexed by i G /. Every bag has a set of 
samples Mi and a set of annotations M . In our application, 
Mi is the group of person tracks appearing in a scene while 
A i can be thought of as a set of sentences specifying who is 
doing what. In the following, we write N = J2iei \Mi\. 

For every sample n G Mi we have a feature vector 
x n G R lxd , and some latent variables detailed next. Every 
sample belongs to a class in V and a class in A. For each 
sample we therefore define a pair of latent variables z n in 
{0, l} lxP and t n in {0, l} lxA indicating to which person 
and action class it belongs. We define A to be a N x d data 
matrix with rows x n , Z is a N x P matrix with person la¬ 
bels in rows z n and T is a N x A matrix with action labels 
in rows t n . The p-th element of a vector z n is written z np . 

Given weak supervision in the form of constraints on Z 
and T (more on these in the next section), we want to re¬ 
cover latent variables z n , t n for every sample x n and learn 
two multi-class classifiers / : R d -> R p and g : R d R A 
(for persons and actions respectively). Because the two sets 
of classes correspond to very different aspects of the data, 
we define two feature maps and ^ that will be respectively 
taken as input for / and g. With a slight abuse of notations, 
we will represent with (j)(X) (respectively 'ip(X)) the matrix 
whose rows are the (j){x n ) (respectively 

2.2. Problem Formulation 

Our problem can be decomposed as a sum of two cost 
functions (for person names and actions) that are linked by 
joint constraints. To avoid repetitions, we will first derive 
equations for one of the two cases only. Let us consider a 
multi-class loss function i : R p x R p —>• R, and denote by 
Cl : F R some regularization function over the set F of 
prediction functions under consideration. We formulate the 
recovery of the latent variables and the construction of the 


classifier as the following optimization problem: 

min ^ Zn ’ f^ x n))) + fi (/) (1) 

’ i£l n£j\fi 

under some constraints defined in section 2.3. We define £ 
to be a square loss, the regularization term U(/) is defined 
by a squared £ 2 norm, and / is a linear classifier: 

f{4>{x n )) = 4>{x n ) w + b , re G M dxP , b G M lxP . 

The optimization problem now becomes: 

min -?-||Z — <p(X)w — b\\ 2 F + Ai Tr(w T w). (2) 

Z.w.b N 

Following [2], we note that (2) admits a closed form so¬ 
lution in w and b for fixed Z. Using this solution, we re¬ 
write (2) as: 

nun Tr(ZZ T A(X, Ai)), (3) 

where A(X, Ai) is a N x TV matrix that depends on the data 
X and the regularization parameter Ai. 

Using (3), we next define a joint optimization problem 
over action labels T and person labels Z as: 

min Tr{ZZ T A(X , Ai)) + Tr(TT T B(X , A 2 )). (4) 

Matrices A , B will be explicitly defined in Section 2.6. 

Note that the above formulation does not contain any 
coupling between 0 and T per se. We will use information 
mined from scripts to couple 0 and T by joint constraints 
as described below. 

2.3. Annotations as Constraints on Latent Variables 

We would like to constrain solutions of our problem by 
coupling person and action labels. We do this using infor¬ 
mation mined from movie scripts. After aligning scripts 
with videos [9], we extract person-action pairs (p, a) and 
their approximate temporal locations. Given a pair (p, a) 
found in the script, we assume a person p performs an ac¬ 
tion a at the corresponding temporal location in the video. 
We model this assumption by constraints defined on the la¬ 
tent variables. To make the best use of the textual data, we 
distinguish three kinds of extracted pairs, (p, a), (p, 0) and 
(0, a), leading to three types of constraints. 

In scene descriptions found in scripts, we observe 
subject-verb pairs and associate those with either (p, a) or 
(0, a) pairs. The distinction comes from the fact that some 
subjects may be pronouns and therefore not designate any 
specific character a priori. 

The (p, 0) pairs come from another source of textual in¬ 
formation: movie scripts contain both scene descriptions 


and dialogues with speaker identities specified. We there¬ 
fore use this information to suggest speaker presence in the 
surrounding video. 

For every person-action pair (p, a ) we construct a bag 
i containing samples Mi corresponding to person tracks in 
the temporal proximity of (p, a). If multiple pairs have sim¬ 
ilar position in time, we group them, producing bags with 
several (p, a) pairs A i. Once the bags are defined, we use 
annotations to constrain the latent variables of person tracks 
in the bag. What we want to model is the following: “if a 
person-action pair is mentioned in the script, it should ap¬ 
pear at least once in the bag”. This can be translated in 
the form of constraints on sums of latent variables of tracks 
within a bag as: 

Vi G I , V(p, Cl) G A i , ^ ^ Z n p tna ^ L (5) 

TlEiAfi 

v(p, 0 ) g a i , 'y ^ z n p ^ 1 , 

neX 

G A i , ^ ^ t na > 1- 
neAfi 

Constraints based on (p, a) provide coupling between the 
two sub-parts of our problem. Pairs (p, 0) and (0, a) de¬ 
fine independent constraints on the person and action latent 
classes respectively. 

Since we have partial knowledge about class member¬ 
ship of samples in bags, our problem is related to multiple 
instance learning (MIL) [2 ]. MIL, however, is not clearly 
defined for the multi-class case. In the binary case it con¬ 
sists in learning a binary classifier given bags containing 
samples of both classes and bags containing only negatives. 
When considering an analogous problem in the multi-class 
case, it is unclear what the bag assumptions would be. 

2.4. Slack Variables 

In practice, person-action pairs in scripts may not al¬ 
ways have corresponding person tracks in the video. This 
can happen due to failures of automatic person detection 
and tracking as well as due to possible mismatches between 
scripts and video tracks. To cope with these issues, we intro¬ 
duce slack variables allowing the constraints to be violated. 
We define a vector £ of length J2 ieI |A^| and rewrite our 
problem as: 

min Tr(ZZ T A(X, Ai)) + Tr(TT r B(X, A 2 )) + k 
s.t. Vi G / , V J G Ai : 

XmG Afi Zn P ^ na — 1 — ^ J ~ {Pi a \ 

< T,neXi Z np > 1 - if J=(p,0), (6) 

> 1 if J = (0,a). 



2.5. Optimization 

When Z and T take binary values, solving the problem 
defined by eq. (6) is NP hard. We thus relax it by consider¬ 
ing real-valued positive matrices Z, T such that Z lp = 1, 
Z > 0 and T 1 A = 1, T > 0. 

The relaxed problem is not jointly convex in Z and T be¬ 
cause of the coupling constraint in eq. (5). Once we fix one 
of the two matrices, the coupling constraint becomes linear 
in the other latent variable. We, therefore, perform a block 
coordinate descent and alternate optimization by solving for 
one of the matrices Z, T while fixing the other. Each of 
the two steps is a convex quadratic program under linear 
constraints since A and B are positive-semidefinite by con¬ 
struction. 

In the first step we freeze the T variable and optimize 
over Z and £. We initialize T with the uniform assignment 
matrix T = Since the two steps are separately 

convex, the initialization of Z does not matter. 

Rounding. Given estimates of real-valued matrices Z 
and T, we have to choose classes for every sample. To 
do so we compute the orthogonal projection according to 
£ 2 norm on the set of indicator matrices Z = {Z E 
{0, l} JVxP | Zip = ljv}: 

argmin \\Z - Z\\ 2 . (7) 

zez 

This amounts to taking maximum values along rows of Z 
and T. For each row the arguments of the maximum de¬ 
fine classes of corresponding samples while the maximum 
values are used as confidence values in our evaluation. 

Relation to Diffrac [2]. Our problem formulation in (4) 
is closely related to the discriminative clustering approach 
Diffrac [2, 17]. When latent classes are treated equally, the 
minimization of a convex relaxation of (4) results in a trivial 
solution [12]. To overcome this issue one can perform a lift¬ 
ing and optimize (4) with respect to the equivalence matrix 
M = ZZ T instead (under a suitable set of constraints). 

Working with M is problematic in our case since our 
constraints in (5) are defined on the elements of Z rather 
than on M. Class-dependent constraints in our case, how¬ 
ever, break the symmetry in class labels and enable (4) to 
be solved directly for Z. In practice we found that modify¬ 
ing the value of 1 to a larger constant on the right sides of 
inequalities (5) leads to a more stable solution of (4). 

2.6. Use of Kernels 

As mentioned in [' ], the optimization problem (3) allows 
the use of kernels. Using the matrix inversion lemma, one 
can derive an expression for A and B that depends only 
on the Gram matrix of the linear kernel (XX T ). We can, 


therefore, replace XX T by the Gram matrix of any kernel, 
yielding in our case: 

a(Ki,\i) = \iH N (h n KiH n + nXxIn) 1 n^, 
£>(^2,A 2 ) = A 2 II/v (Hn^Un + XX 2 In) 1 IIv, 

where II at is the projection matrix II n = In ~ jj In 1jv 
and 

V(i, j) e {1, • • •, N} 2 , (Ki)ij = K f (</>(xi), 

(K 2 )ij = K a (i/>(xi),il>(xj))- 

Kf and K a are the two kernels that we use for faces and 
actions as described in more details in Section 3. 

3. Features and Dataset 

In this section we describe features extracted from the 
text and the video, and give details about the used dataset. 

Text processing. We extract person-action pairs from text 
using a semantic role labeling parser. Semantic role labeling 
consists of identifying arguments (agent, instrument, man¬ 
ner, cause) to a predicate (for example a verb). Intuitively, 
this amounts to answering questions such as “Who” “What” 
“When” “Where” “Why”. Several statistical parsers are 
available on-line. We use SEMAFOR [7], which is trained 
on the FrameNet database [3]. We focus on two frames that 
appear often enough in the script and have an associated 
agent: “ChangePosture” and “SelfMotion”. From each de¬ 
tected occurrence of the frame in the text we use the “agent” 
and the “target verb” as the name and action pair. 

Video features. The aim here is to design a representation 
of video that can be related to the name and action structures 
extracted from the text. This is achieved by automatically 
extracting tracks of people from video. Each person is then 
represented by its face appearance to capture identify and 
motion features to represent the action. See figure 2. 

To extract person tracks, we run the multi-view face de¬ 
tector of [26] and associate detections across frames using 
point tracks in a similar manner to [9, 22]. To represent 
faces we follow [22], and extract facial features and rectify 
each face into a canonical frame using a similarity trans¬ 
formation. We then re-compute facial feature positions in 
the rectified image and extract SIFT descriptors at multiple 
scales from each facial feature. The descriptor for each face 
is formed by the concatenation of all SIFT descriptors. Fi¬ 
nally, each track is represented by the set of descriptors, one 
for each face in the track. 

To represent actions, we compute bag-of-features on 
dense trajectories [25] extracted from each person track. We 
take the trajectories that fall into the spatio-temporal volume 
defined by the upper-body bounding box in each frame. The 



Figure 2: Representing video. Top: face track together 
with extracted facial features. Bottom: Motion features 
based on dense point trajectories extracted from tracked up¬ 
per body bounding boxes. 

upper-body bounding box is defined here by simply extrap¬ 
olating the face bounding-box using a linear transformation. 
This assures that in every frame we have a corresponding 
face as well as an upper-body region. Our discriminative 
cost function allows the use of kernels. For face tracks, we 
follow [22] and use the sum of “min-min kernels” computed 
separately for each facial feature as well as frontal and pro¬ 
file faces. This results in a total of 38 face track kernels 
(24 for frontal features and 14 for profile features) that are 
summed with uniform weights. For action descriptors we 
use the exponentiated chi-square kernel [25]. 

Dataset. We report results for movies Casablanca and 
American Beauty. For both movies we extract person tracks 
and associated descriptors. We discard person tracks with 
unreliable facial features based on the landmark localization 
score. For Casablanca, we obtain 1,273 person tracks con¬ 
taining 124,423 face detections while for American Beauty 
we use 1,330 person tracks containing 131,741 face detec¬ 
tions. 

By processing corresponding movie scripts, we extract 
17 names for the main characters in Casablanca and 11 
names for the main characters in American Beauty. For 
each movie we select two most frequent action classes, i.e., 
walking, sit down for Casablanca and walking, open door 
for American Beauty. For Casablanca we obtain 42 ac¬ 
tion/name pairs and 359 occurrences of names with no as¬ 
sociated actions. For American Beauty the corresponding 
numbers are 31 and 330, respectively. The dataset is avail¬ 
able at [1]. 

To explicitly model non-named characters in the movie 
(side characters and extras) as well as non-considered ac¬ 
tion classes we introduce an additional “background” class 
for both faces and actions. We collect background exam¬ 


ples as follows. For faces, we collect additional 300 ran¬ 
dom faces from the Labeled Faces In The Wild dataset [15]. 
For actions, we randomly sample 500 person tracks from 
the Hollywood2 dataset [2( ] using the corresponding movie 
scripts to discard actions considered in this work. For all 
“background” samples, we constrain latent variables to take 
values corresponding to the “background” class. We found 
that including this additional data helps resolving confusion 
in label assignment for our target classes. 

4. Experiments 

In this section we experimentally demonstrate the bene¬ 
fits of the proposed approach. We first test the sensitivity 
to parameter choices in a controlled character identification 
setup. Second, we show that even for learning names alone 
(without actions) the proposed method outperforms other 
state-of-the-art weakly supervised learning techniques de¬ 
signed for the same task. Finally, we demonstrate benefits 
of learning names and actions jointly compared to resolving 
both tasks independently. 

Learning names: controlled set-up. Here we wish to as¬ 
sess the sensitivity of the proposed method to the follow¬ 
ing four important parameters: the number of bags |/|, the 
number of classes P, the number of samples per bag |A41 
and the number of annotations per bag |A*|. We will use 
real data - 1,273 face tracks and their descriptors from the 
movie Casablanca - but group the tracks into bags in a con¬ 
trolled manner. Each track is labeled with a ground truth 
name from the set of 18 main characters (or other). To cre¬ 
ate each bag, we first sample a track from a uniform distri¬ 
bution over characters and then complete the bag with up 
to \J\fi \ tracks by randomly sampling tracks according to the 
true distribution of the characters in the movie. Each bag is 
annotated according to the first sample. Given this data, we 
solve the sub-problem related to faces, i.e. no joint action 
labels are used in this experiment. 

As discussed in Section 2, each face track is assigned 
to a class by maximizing the rows of Z. Classified face 
tracks are then sorted by their confidence values and the per¬ 
centage of correctly classified tracks (i.e., the per-sample 
accuracy) is evaluated for each confidence value. Follow¬ 
ing [9, 22 ] we measure performance by plotting a curve of 
per-sample accuracy vs. proportion of labeled tracks. Ide¬ 
ally, the accuracy would be one for all confidence values, 
but in practice the accuracy drops for samples with lower 
confidence. We illustrate results for different bag layouts in 
Figure 3. 

Comparison with other weakly supervised methods. 

Here we compare our method with other weakly supervised 
face identification approaches. We use the code adapted 
from [2 ] and an on-line available implementation of [6]. 
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■ 5 samples, AP=0.49 

■ 6 samples, AP=0.52 


proportion of labelled tracks 



(a) (b) (c) (d) 

Figure 3: Performance for different bag layouts in a controlled set-up. (a) First, we vary the number of bags while fixing 
3 samples and 1 annotation per bag, and the number of classes to 5. As expected, performance improves with more bags, (b) 
Keeping 150 bags in total, we increase the number of classes. The effects of this modification are mixed. By adding more 
classes, the problem is harder but the per bag confusion is smaller, (c) Keeping 7 classes, we increase the number of samples 
per bag showing that more samples per bag increase confusion resulting in a lower performance, (d) Keeping 5 samples per 
bag, we increase the number of annotations per bag, clearly showing the benefits of having more annotations. 


We run all methods on 1,273 face tracks from Casablanca 
and 1330 face tracks from American Beauty using noisy 
name annotations obtained from movie scripts. To have 
a fair comparison, no action labels are used. While [i ] 
and [22] have been evaluated on television series, here we 
address a more challenging setup of full-length movies. 
First, the training data within a film is limited as it is not 
possible to harvest face tracks across multiple episodes as 
in TV series. Second, the cast of characters in a film is often 
larger than in TV series with many additional extras. Third, 
films often employ a wider set of cinematographic tech¬ 
niques compared to often simpler structure of a TV show 
with many close-ups and “shot-reverse shot” dialogues. 

Comparative results for the two movies in Figure 4 
demonstrate superior performance of our method. The 
lower performance of [21 ] can be explained by its de¬ 
pendency on the visual speaker identification. While our 
adaptation of the code obtained from the authors of [22] 
worked well on their data, we found that the speaker detec¬ 
tion achieved only 64.2% and 50.2% accuracy (with about 
25% speaker labeled tracks) on Casablanca and American 
Beauty, respectively. The lower accuracy, compared to the 
accuracy of more than 80% on the TV series data from [22], 
could be possibly due to the challenging illumination con¬ 
ditions with strong shadows present in the two films. The 
approach of [6] assumes that correct labels are included into 
the set of “ambiguous” labels. This assumption is often vi¬ 
olated in movies as side characters and extras are often not 
mentioned in the script. In contrast, our approach suffers 
less from this problem since (a) it can handle multiple an¬ 
notations for bags of multiple tracks and (b) the noise in la¬ 
bels and person detections is explicitly modeled using slack 
variables. 

Learning names and actions. We next evaluate benefits 
of learning names and actions jointly. This is achieved by 



(a) Casablanca (b) American Beauty 

Figure 4: Results of automatic person naming in movies. 
Our method is compared with weakly supervised face iden¬ 
tification approaches of Cour et al. [6] and Sivic et al. [22]. 

first learning the name assignments Z for all tracks. The 
name assignments are then fixed and used as additional con¬ 
straints when learning the likely action assignments T for 
each track. While this procedure can be iterated to improve 
the assignment of actor names with the help of estimated ac¬ 
tion labels, we found that the optimization converges after 
the first iteration in our setup. 

The distribution of action classes in our data is heavily 
unbalanced with the “background” class corresponding to 
more than 78% of person tracks. We therefore evaluate the 
labeling of each target action in each movie using a stan¬ 
dard one-vs-all action precision-recall measure. We com¬ 
pare the following methods. Names+Actions corresponds 
to our proposed method of learning person names and ac¬ 
tions jointly. No Names uses constraints on actions only 
without considering joint constraints on actions and names. 
True Names+Actions uses the ground truth person names 
as constraints on actions instead of the automatic name as¬ 
signment. This provides an upper bound on the action clas¬ 
sification performance provided perfect assignment of per- 
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Figure 5: Results of action labeling in movies Casablanca and American Beauty. See Section 4 for more details. 


son names. Finally, we evaluate two “dummy” baselines 
which blindly assign action labels based on person names 
and person-action pairs obtained from scripts. The purpose 
of these baselines is to verify that visual action classification 
improves the performance. Names+Text learns face assign¬ 
ments for each person track and assigns action labels using 
person-action pairs. True Names+Text assigns action la¬ 
bels based on person-action pairs and ground truth person 
names. This baseline, hence, does not “look” at image pix¬ 
els at all. Note that the last two baselines produce a single 
point on the precision-recall plot as no confidence values 
are available when transferring action labels from scripts. 

Precision-recall plots for the target action classes in two 
movies are shown in Figure 5. We first observe that our full 
method (blue curves) outperforms the weakly supervised 
learning of actions only (green curves) in most of the cases. 
This shows the benefit of learning actions and names jointly. 
As expected, action classification can be further improved 
using ground truth for name assignments (red curves). 

For the frequent action walking for which many person- 
action constraints are available in scripts, automatic person 
naming in our method provides a large benefit. However, 
even with ground truth face assignments the action classi¬ 
fication performance is not perfect (True Names+Actions). 
This is likely due to two reasons. First, the ambiguity in the 
weak supervision is not reduced to zero as a single charac¬ 
ter may do several different actions in a single clip (bag). 
Second, the current action representation has only limited 
discrimination capabilities. 

Recognizing less frequent actions sit down and open 
door appears to be more difficult. While several examples 
are ranked high, all methods suffer from a small number of 
available person-action constraints. In addition, a signifi¬ 
cant portion of these constraints is incorrect. Incorrect con¬ 
straints often occur due to the failure of face detection as ac¬ 
tors often turn away from the camera when sitting down and 
opening doors. To explicitly quantify the loss due to failures 
of automatic person tracking, we have manually annotated 
person tracks in the movie Casablanca. The performance of 
our full method is significantly improved when run on cor¬ 


rect person tracks yielding AP=0.36 and AP=0.63 for the 
sit down and walk actions, respectively. This emphasizes 
the need for better automatic person detection and tracking 
methods. Qualitative results for automatic labeling names 
of actors and actions using our method (Names+Actions) 
are illustrated Figure 6. More results are available at [1]. 

5. Conclusion 

We have developed a new discriminative weakly super¬ 
vised model jointly representing actions and actors in video. 
We have demonstrated the model can be learnt from a fea¬ 
ture length movie together with its shooting script, and have 
shown a significant improvement over other state-of-the-art 
weakly supervised methods. As actions are shared across 
movies, applying the model over multiple movies simulta¬ 
neously opens-up the possibility of automatically learning 
discriminative classifiers for a large vocabulary of action 
classes. 
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